Augmented reality system for visualization of traffic information in a transportation environment

ABSTRACT

An augmented reality visualization framework for precision traffic analysis combines traffic data with content representative of physical-world characteristics, such as conditions in an intersection or roadway in which traffic activity occurs, and images of the intersection and roadway itself. The framework includes a visualization platform that enables display of the combined traffic data and content representing physical-world characteristics, and continual augmentations of such information in response to adjustments directed by users or automatically detected from actions such as user movement, other user manipulation, or movement of the visualization platform itself. The framework enables traffic applications such that a user can, when the platform is interfaced with intersection or roadway equipment, adjust machine activity or signal timing based upon the visualizations or data presented via the visualization platform. The visualization platform may also be configured to present the traffic data, and the content representative of physical-world characteristics, in a virtual reality setting.

CROSS-REFERENCE TO RELATED PATENT APPLICATION(S)

This patent application claims priority to U.S. provisional application62/631,712, filed on Feb. 17, 2018, the contents of which areincorporated in their entirety herein. In accordance with 37 C.F.R. §1.76, a claim of priority is included in an Application Data Sheet filedconcurrently herewith.

FIELD OF THE INVENTION

The present invention relates to augmented reality systems, and tovehicular traffic management in transportation systems. Specifically,the present invention relates to a system and method for combiningtraffic data with actual-world reality, such as an image or video of anintersection or roadway in which a vehicle or other object representedby or relative to the traffic data is shown, to provide an augmented,virtual, or mixed-reality environment for precision traffic managementapplications.

BACKGROUND OF THE INVENTION

Implementations of augmented and/or virtual reality in the field oftransportation systems are presently very limited. Applications existthat allow users to view data that is overlaid on a map of a particulargeographical area, for example on mobile devices. Overlaid data mayinclude intersection or roadway-specific, activity-specific, andmachine-specific data, and users can tap or select particularinformation, or particular areas of an intersection or roadway, forfurther detail. However, development of these applications isrudimentary, and has not reached a stage to where cities, counties,states, and other responsible entities can view visualizations of anintersection or roadway that are augmented with such data rather thansimple overlays, for example using a wearable device. Additionally,development of these applications has not reached a stage wherecontinuous augmentations of sophisticated traffic information arepossible, nor have they reached a stage where they can be adjusted basedon real-time user input.

It is to be understood that virtual reality, or VR, refers to thecreation of a virtual world that users can interact with, and in whichusers would find it difficult to tell the difference from what is realand what is not. Meanwhile, augmented reality, or AR, is the blending ofvirtual reality and real life, in which images may be created withinapplications that blend in with contents in the real world. With AR,users are able to interact with virtual contents in the real world, andare able to distinguish between the two. Virtual reality and augmentedreality are similar in the immersion of the user, though this isaccomplished in different ways. With AR, users continue to be in touchwith the real world while interacting with virtual objects around them.With VR, the user is isolated from the real world while immersed in aworld that is completely, or almost completely, fabricated.

BRIEF SUMMARY OF THE INVENTION

The present invention provides a framework within which trafficinformation is presented to users in an augmented or mixed-realitysetting. Such a framework allows users to view or display selectedinformation in relation to content representing a physical-worldcharacteristic, such as an actual intersection or roadway for which suchtraffic information is of interest. One or more data processingtechniques enable the augmented combination of traffic information andthe content representing the physical-world characteristic.

Both augmented and virtual reality systems can be used to supplementexisting reality representations of the actual world, usually byincorporating information and graphics into a captured image or videostream or a transparent display surface. Such systems can be used, inconjunction with their internal components, to display traffic,environmental and situational data, markers, and visualizations forareas of interest (AOIs) and points of interest (POIs). Varioustraffic-related data elements may be combined together, through dataprocesses or through transparencies or blending techniques, along withimagery or video of the environment surrounding the display apparatus orvisualization platform. These traffic-related data elements arecontinuously augmented and adjusted according to the visualizationplatform's capabilities (e.g., movement, tilting, panning, zooming,etc.) and/or by user movement or manipulation of the visualizationplatform.

It is therefore one objective of the present invention to provide asystem and method of displaying traffic information to a user in anaugmented reality system. It is another objective of the presentinvention to present such data in combination with one or more images ofan actual intersection or roadway on augmented reality-enabled devices.It is yet another objective of the present invention to present advancedalgorithms and traffic analytics to users as they traverse anintersection or roadway, using or wearing augmented reality-enableddevices. It is a further objective of the present invention to enableusers to interface with intersection or roadway equipment such astraffic cabinets and signaling systems by manipulating an augmentedreality system relative to an intersection or roadway in which theintersection or roadway equipment is utilized.

Other objects, embodiments, features, and advantages of the presentinvention will become apparent from the following description of theembodiments, taken together with the accompanying drawings, whichillustrate, by way of example, principles of the invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and together with the description, serve to explain theprinciples of the invention.

FIG. 1 is a block diagram illustrating system components in an augmentedreality visualization framework for traffic applications according toone embodiment of the present invention; and

FIG. 2 is a flowchart of steps in an augmented reality visualizationframework for traffic applications according to one embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description of the present invention, reference is madeto the exemplary embodiments illustrating the principles of the presentinvention and how it is practiced. Other embodiments will be utilized topractice the present invention and structural and functional changeswill be made thereto without departing from the scope of the presentinvention.

The present invention is an augmented reality visualization framework100 for traffic and transportation applications. FIG. 1 is a systemicdiagram of components in such an augmented reality visualizationframework 100 for traffic and transportation applications, in whichcontent representative of a physical-world characteristic 102 isaugmented with input data 110 representative of traffic activity 111 togenerate an augment reality visualization 123. In one aspect thereof,the present invention is an application of traffic data elements in thistraffic activity 111 to an augmented or mixed reality setting, where auser is able to view selected information in relation to an actualintersection or roadway for which such information is of interest, andto adjust that information, within the augmented reality visualization123. In another aspect thereof, the traffic and roadway augmentedreality visualization framework 100 includes a visualization platform120, within and upon which such an application of traffic data elements111 to an augmented or mixed reality setting is performed to generatethe augmented reality visualization 123.

The visualization platform 120 may include a display apparatus 122 uponwhich a user is capable of viewing the combination of traffic dataelements 111, and content representing physical-world characteristics102 related to such information, such as an intersection 104 or roadway106 in which a vehicle or other object is shown or in which trafficsignaling equipment 119 is operating, together comprising atransportation environment 108. A display apparatus 122 may include awireless augmented or virtual reality headset or other similarhead-mounted (or other wearable) gear capable of projecting imagesthrough one or more lenses for viewing by a wearer or user. The displayapparatus 122 may be any article or device configured to allow a user toview the application of traffic data elements 111 as an overlay on thephysical-world characteristic(s) 102 representing the transportationenvironment 108, as well as manipulate the traffic data elements 111displayed in some manner.

In one example in which the present invention is implemented, a uservisiting the transportation environment 108 may direct theirhead-mounted display at the scene comprising intersection 104 or roadway106, and select any number of overlays to view related to trafficactivity in that scene. The user may also adjust opacity levels for eachoverlay, and see areas of interest (AOIs) and/or point of interest(POIs) displayed on top of the intersection 104 or roadway(s) 106visible through the display. In another example, a user visiting a sceneof the transportation environment 108 comprising a traffic intersection104 or roadway 106 may direct a mobile device to capture a video streamof the scene or environment (e.g., the intersection 104 or roadway 106)in front of the device's camera, select any number of overlays to view,adjust opacity levels for each overlay, and see AOIs and/or POIsoverlaid on the intersection 104 or roadway(s) 106 visible through thedisplay.

In yet another example, an operator (the user) may view a projection onor within windows of a machine that displays, among other selectableoverlays, the outline of transportation environment 108, the area of thetransportation environment 108 already covered by the machineryimplement or other cooperating implements, the location of othermachines in the transportation environment 108, and any otherinformation to help the operator visualize and locate other machines orobjects in the scene. This may additionally include information toassist in coordinating joint operations involving traffic equipment(e.g., setting up a work zone), and planning of future operationsinvolving such equipment (e.g., turns to avoid obstacles, visualizingthe width of lanes).

Other examples of a display apparatus 122 embodying the visualizationplatform 120 on which the augmented reality visualization 123 arerendered include a mobile phone or other computing device, such astablet or laptop computer, holographic displays such as images shown onglass or a window (e.g., a projection within the glass), wearables suchas Google Glass and VR helmets [e.g., Oculus Rift, HTC Vive, etc.], andMicrosoft HoloLens, etc. A display apparatus 122 may further include aheads-up display (e.g., projection onto glass), a pop-up display, or a3-D projection. It is therefore to be understood that the presentinvention is not to be limited to any one type of display apparatus 122mentioned herein, and that any type of display may be configured for thetraffic and roadway augmented reality visualization framework 100 fortransportation applications of the present invention.

The present invention is embodied in one or more systems and methodsthat enables such combinations of traffic data elements 111 withphysical-world characteristics 102 representing a transportationenvironment 108, such as imagery and mapping of geographical areasrelated to these traffic data elements 111. These systems and methodsare performed, in one aspect of the present invention, in a plurality ofdata processing modules 130 that are components within a computingenvironment 140 that also includes one or more processors 142 and aplurality of software and hardware components. The one or moreprocessors 142 and plurality of software and hardware components areconfigured to execute program instructions or routines to perform thecomponents, modules and data processing functions described herein, andembodied within the plurality of data processing modules 130.

These processors 142, components and data processing functions togetheraccomplish the combination of traffic data elements 111 withphysical-world characteristics 102 for visualization in an augmented ormixed reality setting according to the present invention. This includesthe capture and/or display of content as input data 110 representingtraffic-related activity 111 relative to a physical-world characteristic102, using a display apparatus 122 or other visualization platform 120.Such content/input data 110 may include video 112 of an intersection 104or roadway 106 comprising a transportation environment 108, and shown orstreamed on a screen of a device such as virtual reality headset. Such avideo stream 112 may be captured by the device itself, or acquired froman external source. The video stream 112 may therefore also be areal-time representation of the intersection 104 or roadway 106 andconditions currently being experienced therein.

The plurality of data processing modules 130 may therefore include adata ingest module 132 configured to ingest, capture, request, receive,or otherwise obtain the input data 110. Several other modules 130 may beincluded as further described herein, including a module 134 forconfiguring and modifying data relative to the physical-worldcharacteristic 102 for display on the visualization platform 120, and amodule 136 for configuring and modifying traffic data 111 and otherinput data 110 for display on top of the data relative to thephysical-world characteristic 102 on the visualization platform 120. Themodules 134 and 136 may perform this configuration and modification oftraffic data 111 by one or more of filtering, ordering, adjusting, orotherwise transforming according to, for example, the view of the user,and manipulation of the visualization platform 120 by the user. Othercomponents 130 include a blending module 137 that applies one or moretechniques for combining the various types of input data 110 with thedata relative to the physical-world characteristic 102, and a trackingmodule 138 that enables the framework 100 to continually augment theinformation provided to the user via the visualization platform 120.

The input data 110 also includes the detection of a geographic position114 of a user or the visualization platform 120, as well as movement116, such as direction, motion, tilt, or other motion of the user orvisualization platform 120. Therefore, one or more global positioningsystem (GPS) components 115 may be included within the presentinvention, including a GPS-enabled receiver configured to correlate suchmovement 116 with information related to both the traffic activity 111and intersection 104 or roadway 106 data to be displayed by computingthe user's precise position and orientation in the transportationenvironment 108 relative to the Earth. The GPS-enabled receiver of theGPS components 115 may thus extract and determine the geographicallocation 114 of the user relative to that of the transportationenvironment 108 for which information is sought from GPS data points.

The present invention also includes sending a request for nearby trafficactivity information 111, such as traffic, environmental and situationalareas of interest (AOIs) 117 and places of interest (POIs) 118 based onsearch criteria. Such search criteria may be defined by the useraccording to the type of information sought. GPS data points regardingthe location 114 of the user or the visualization platform 120 itselfalso allow for obtaining or ingesting the correct traffic information111 requested. A request may be initiated by the user or may beautomatically generated, for example as a result of informationextracted from such GPS data points or by detected action in the user orvisualization platform 120. A further step of receiving such nearbytraffic, environmental and situational AOIs 117 and POIs 118, inresponse to the request, is another function performed in the presentinvention.

The present invention further includes visually augmenting the capturedand/or displayed video stream 112 with the information received for thenearby AOIs 117 and POIs 118. The visual augmentation may occur in anynumber of ways. For example, augmentations may be presented in the formof colorized, three-dimensional regions overlaid on top of geographicalfeatures for the intersection 104 or roadway 106 being viewed, floatingimagery (small popup imagery, thumbnails, previews, etc.), flags orother markers, and icons and other iconography. Examples of colorized,three-dimension regions include an exaggerated topological color-mapoverlaid on a region of interest at the level of an intersection 104 orroadway 106, such as to show the minute differences in elevation for usewhile performing operations in the transportation environment 108. Otherexamples of colorized, three-dimension regions include maps and maps ofvehicle placement zones. The information received for the nearby AOIs117 and POIs 118 may also be filtered, ordered, adjusted, or transformedbased upon user input.

The present invention also includes visually augmenting the capturedand/or displayed video 112 with information from signaling equipment 119at or near an intersection. Signaling equipment 119 may be associatedwith a system of traffic lights or other equipment 108, a traffic signalcontroller 158, with one or more signs posted at the intersection 104,or traffic notification systems, such as those that provide differenttypes of notifications to certain roadway users such as pedestrians andbicyclists.

Information presented in visual augmentations or visualizations may alsoinclude scouting reports (representing, for example, locations scouted,areas of concerns, noteworthy items, etc.), objects located inintersection or roadways (such as equipment or machines, debris,obstacles, wet spots, etc.), prior tracks of equipment and unintendeditems in the intersection 104 or roadway 106.

Visual augmentations or visualizations may also include live-streamed,retrieved, raw, processed, analyzed, transmitted, collected, and/orcombinations of one or more of the aforementioned, and of additionaldata elements such as UAV sensor data, other autonomous vehicle data,satellite sensor data, vehicular yield data, topological data,temperatures, etc., as well as specific maps showing roadway properties,maps or datasets, and as-applied maps or datasets, and any otheranalyzed data sets.

Traffic-related activity data 111 for visualizations may come from manydifferent sources. Examples of such sources include sensors within thevisualization platform 120 or its host, previously-stored information,data acquired directly by traffic equipment such as machines, andvehicles (in-intersection or roadway or otherwise), and/or sensors andtheir related systems (associated with such as equipment or otherwise),and both publicly-available and privately-acquired data sources. Inputdata 110 may also be provided, as noted above, by signaling equipment119 located at or near a traffic intersection 104, either associatedwith a system of traffic lights, with one or more signs posted at theintersection 104, or with other traffic notification systems.

There are many other types of traffic activity data 111 that may beutilized to augment a display of a physical-world characteristic 102 orsetting. Such data 111 may include a color-map overlaid on a region ofinterest, a vehicle yield map, a work management zone, special eventinformation, current or expected traffic signal timing, and one or moreof vehicle, bicycle, pedestrian, or object counts in the intersection104 or roadway 106, or weather or pavement conditions 125, eithercurrently being experienced or over a period of time. Such weather orpavement conditions 125 may include temperature and moisture at variouslocations. Still other types of traffic activity data 111 may includevehicle density, vehicle velocity, and vehicle or object classificationand identification, for example as a vulnerable roadway user (such as apedestrian or bicyclist).

Many other components of a visualization platform 120 within theframework 100 are possible and within the scope of the presentinvention. The visualization platform 120 may include a display orscreen, and one or more input devices that are capable of acquiringinput data 110, such as relative-position sensors and cameras, anaccelerometer, a compass, and as noted above, one or more GPS components115. Other hardware that may be included in the visualization platform120 are memory components or units, one or more computer processors,data storage components, a communications interface, and a graphicsaccelerator. One or more projectors may also be included to convey videostreams and incoming data on the display apparatus 122.

The traffic and roadway augmented reality visualization framework 100for transportation applications of the present invention analyzes inputdata 110 and traffic activity information 111 for presentation in manydifferent ways, and the outcome of these analyses may be considered asoutput data 150 of the framework 100. As an example, if an agency hastwo or more years' worth of data for vehicular activity in a particularintersection or roadway, one data processing function in the presentinvention might be to produce as output data 150 a multi-yearperformance map 151 for visualization on a view of the scene of anactual intersection 104 or roadway 106. A variation on such aperformance map 151 might be a map depicting areas that significantlydeviated from the average (mean) in one or more years. Yet another mapdisplayed might be one that depicts variance or mean square error, todepict areas of a transportation environment 108 that have experiencedsignificant performance variations year to year.

Many other examples of temporal analytics of traffic activity data 111such as the multi-year performance map 151 and generally output data 150are possible and within the scope of the present invention. In oneexample, the framework 100 may be configured to display an amount ofvehicle speed variability from one time period to another. In anotherexample, the framework 100 may be configured to display an analysis ofnormalized movement over a locally-temporal movement, indicating wheretoo many vehicles may be affecting congestion on a roadway 106.

Output data 150 may also include specific displays of information tofacilitate transportation planning and decision-making. One such outputmay be current volume at the intersection 104 or roadway 106 againstaverage volume, such as for example a 90-day moving average or anaverage of the last x weeks for a current time of day, day of the week,or other time period (such as monthly, or for the last one, three, orfive years data as a further example). Another exemplary output may be acurrent queue length against an average queue length, again such as a90-day moving average or average of the preceding x weeks for a currenttime of day, day of the week, or other time period (such as monthly, orfor the last one, three, or five years data as a further example).

Other displays of output data 150 may include generating a display ofcurrent volume plotted against typical volume weather for the currentday of year, and current queue length plotted against typical weatherfor the current day of year. Traffic volume during special events(sports, concerts, etc.) may be displayed as compared to current trafficvolumes. Additionally, before-and-after information based on changes tosignal timing or changes to approach layout (changing the number of leftturn lanes for example) or due to construction (increasing approachsize) may also be generated and displayed on the visualization platform120, as well as before-and-after information based on a constructionproject (such as for example a new shopping mall, office building, ormulti-unit residential building).

Combinations of traffic activity data 111, and content related to aphysical-world characteristic 102 (as well as continuous and adaptiveaugmentations of such combinations) may be further performed usingadditional data processing functions. For example, transparency orblending techniques may be applied to one or more data sets forgraphical presentation on the display apparatus 122 of the augmentedreality visualization framework 100. In the augmented realityvisualization framework 100 of the present invention, whether a user ison the edge of an intersection 104 or roadway 106 or within anintersection 104 or roadway 106, the user is able to “look” around andsee the above data overlaid on the actual transportation environment 108(or more than one layer of data at a time, with or without sometransparency for the layers).

It should be noted that in many cases, just having the data is notalways helpful—it may need to be acted upon. For example, in an analysisof normalized movement over a locally-temporal movement that indicateswhere too many vehicles may be affecting congestion, a user standing intransportation environment 108 be able to immediately diagnose and takeaction about the diagnosed issues, for example by operating equipmentwhile using a visualization platform 120 as in the present invention toidentify trouble spots. Output data 150 may therefore comprise one ormore actions 152 that a user takes in response to data analytics withinthe framework 100. There are many examples of where a physical-worldoutcome of data analytics using a visualization platform 120 in theframework 100 of the present invention may be realized.

In one such example, output data 150 may be used to identify possibleproblems with either detection quality or controller response, andtherefore one action 152 is initiating a change in traffic datadetection paradigms 153 and/or adjustments in traffic signal timing andcontrol 154 at a transportation environment 108. Such an action 152 maybe determined, for example, by evaluating the current versus normalcycle time data for a traffic phase, or by evaluating historic trafficvolumes but based under current or forecasted weather conditions withinthe visualization platform 120. Examples of adjusting traffic signaltiming and control 154 include implementing alternate signal timing orphase patterns at an intersection 104 or roadway 106. Historic versuscurrent volumes of traffic activity by vehicle classification (forexample, motorcycles, passenger cars, trucks, busses, etc.) may also beused implement alternate signal timings or phase patterns as actions 152taken as a result of analytics using the visualization platform 120.

In another example, output data 150 may include adjusting traffic signaldilemma zone timing 155 (in conjunction with alternate signal timing orphase patterns) specifically in relation to the speed of vehicles asthey approach an intersection 104 or roadway 106, or vehicular speed ingeneral, within a transportation environment 108. Output data 150 mayalso include adjusting work zone or special event operations 156. Forexample, if a user diagnoses that traffic activity is negativelyimpacted by a work zone or special event (or vice versa) in thetransportation environment 108 and this can be remedied, the presentinvention may generate an instruction to adjust one or more of signaltiming, phase cycles, machinery operation, work zone/special planning,or take other remedial action.

In a further example, output data 150 may include adjusting trafficsignal timing and control 154 (such as implementing alternate signaltiming or phase patterns) specifically in relation to vulnerable roadusers (VRUs) such as pedestrians, cyclists, and motorized scooter users,by analyzing historic volumes for such VRUs in comparison with currentvolumes. Therefore, one additional responsive action 152 is anadjustment 157 to the signal timing and control for vulnerable roadusers.

In still another example, output data 150 may comprise a signal orinstruction to a traffic signal controller 158 itself. The framework 100may therefore be configured to communicate directly with traffic signalcontrollers 158 as a result of user manipulation of the visualizationplatform 120.

The framework 100 is also capable, as noted throughout, of continualaugmentation. For example, if the perspective of the visualizationplatform 120 is altered (via movement, tilting, panning, zooming, etc.),the visualization itself will be adjusted to compensate for thedifference of perspective. Additionally, the augmented realityvisualization framework 100 may be configured to track the intersection104 or roadway 106 in the view of the user wearing headgear or otherdevice, or to track orientation and movement of a user using a mobiledevice while looking at data overlaid on a rendering of thetransportation environment 108, and adjust data displayed accordingly.Therefore, the framework 100 is responsive to both user interaction andmovement vis-à-vis the visualization platform 120.

Regarding tracking of orientation or movement of, or manipulation by, auser, the framework 100 may be configured to response to user activityin many ways. For example, when a user manipulates the display of avisualization platform 120 by touching, tapping or clicking on a trafficsignal, the framework 100 may provide different types of data such asarrivals on red, arrivals on yellow, arrivals on green, and percentagesfor each, to indicate an efficiency of approach and level-of-servicescores. In another example, a user may touch, tap or click on a safetyicon on the display apparatus 122 to provide data on red light runners,number of vehicles detected in a dilemma zone, counts of vulnerable roadusers, work zone management data such as number of workers present andhours of planned maintenance activity, emergency vehicle and/or policeactivity, or other safety concerns. The framework 100 may also respondby using zooming functions to focus data displayed, for example onvolume data at a stop bar, or advanced approach speed.

Regarding tracking of an intersection 104 or roadway 106 in the view ofthe user, the framework 100 may be configured so that when a user isviewing two approaches at an intersection 104 simultaneously, theframework 100 initiates a display of output data 150 representingtraffic in-flow and out-flow volume between the two approaches.Similarly, when the user is viewing both the approaching and departingtraffic on an approach at the intersection 104, the framework 100initiates a display of output data 150 representing traffic in-flow andout-flow volumes. The framework 100 may also respond by using zoomingfunctions to focus data displayed based on tracking of the user's view,for example by zooming into a stop bar and displaying stop bar volumedata, or advanced approach speed relative to a position of the roadway106 tracked in the user's view.

Additionally, it should be noted that user interaction may be bothverbal and physical. The visualization platform 120 may therefore beresponse to both verbal commands by a user, and tactile/touch input fromthe user, as well as by movement, tilting, panning, zooming, and othermotion. For example, the user may speak a command that instructs theframework 100 to zoom into the stop bar, and further, to display stopbar volume data. The framework 100 may alternatively automaticallydisplay stop bar volume data in response to a command to zoom into thestop bar. As a further example, the user may speak a command thatinstructs the framework 100 to zoom into a bicycle lane or crosswalk,and further to display vulnerable road user volume data. The framework100 may alternatively automatically display such VRU volume data inresponse to a command to zoom into the bicycle lane or crosswalk.

The physical-world characteristics 102 combined with traffic dataelements 111 may also include weather-related conditions affectingtransportation activity in a particular intersection 104 or roadway 106.Data representative of weather-related conditions may include recent andcurrent intersection or roadway-level weather data and short-termweather forecast data, and may further include historical, predicted,and other weather information, from many different sources as notedfurther herein. Recent and current intersection or roadway-level weatherdata may represent in-situ or remotely-sensed observations for aparticular intersection or roadway, and may be derived from or provideddirectly via one or more crowd-sourced observations in the addition tothe sources mentioned herein. The data representative of weather-relatedconditions may be ingested into the present invention in raw,unprocessed form, or as processed data in the form of modeled, predictedor forecasted weather data over particular periods of time, such asshort-range weather predictions and long-range, extended, and/ordynamical weather forecasts. Other sources of data representative ofweather-related conditions include the output from numerical weatherprediction models (NWP) and/or surface networks, and may be combinedwith data from weather radars and satellites to reconstruct the currentand near-term forecasted weather conditions on any particular area to beanalyzed. Additionally, internally or privately-generated “mesoscale”NWP models developed from data collected from real-time feeds to globaland localized observation resources may also be utilized. The presentinvention may be configured to ingest data from all types of NWP models,regardless of whether publicly, privately, or internally provided ordeveloped.

FIG. 2 is a flowchart illustrating steps in a process 200 for performingthe augmented reality visualization framework 100 for traffic andtransportation applications, according to certain embodiments of thepresent invention. Such a process 200 may include, as noted above, oneor more functions, mathematical models, algorithms, machine learningprocesses, and data processing techniques for the modules 130 thatperform augmentation of content representative of physical-worldcharacteristics 102 relative to an intersection 104 or roadway 106 andinput data 110 representing traffic data 111 and other informationwithin such a framework 100, and for the various analytical approachesapplied within each module 130.

The process 200 begins at step 210 by initializing content representingphysical-world characteristics 102 that include, as noted throughout, anintersection. 104, a roadway 106, or conditions affecting anintersection or roadway where traffic activity occurs. This content iscaptured by one or more cameras present at the transportationenvironment 108, and may be initialized before or after ingest by thedata ingest module 132. This content includes one or more video imagesof the one or more physical-world characteristics 102. In addition tovideo data 112, other types of data representing the intersection 104,roadway 106, or conditions affecting them may also be used, such as forexample satellite imagery. At step 220, this video data is rendered fordisplay on the visualization platform 120, by applying the dataprocessing functions performed within the module 134 configured tomodify and configure the video data 112 for such a visualizationplatform 120.

At step 230, the process 200 detects location 114 and movement 116 ofthe visualization platform 120. This may occur for example from usermovement while wearing an augmented reality or virtual reality headset,or other motion or interaction initiated by the user. Locationinformation may be provided by GPS components 115 as noted above, atleast some of which may be located in the visualization platform. 120itself. This information is used to augment the video data 112 shown onthe visualization platform 120, and also to prepare the framework 100for the ingest of traffic activity data 111 and other relevantinformation to overlaid on the video data 112 representing thephysical-world characteristics 102.

Once location 114 and/or movement 116 (or other manipulation of thevisualization platform 120) has been detected, the process 200 continuesat step 240 by requesting traffic activity data 111 for the relevantintersection 104 or roadway 106, or that specifically requested by theuser. This traffic activity data 111 is then provided to the module 136that modifies and configures this information for the visualizationplatform 120, and at step 250 this information is overlaid on the videodata 112 on the visualization platform 120 for viewing by the user. Atstep 260, the traffic activity data 111 may be blended with the videodata 112 (together with any other ingested information) using theblending module 137 to further augment the display of informationrelated to the physical-world characteristics 102.

The framework 100 continually tracks the user's view of the intersection104 or roadway 206 at step 270, as well as the user's orientation andmovement 116. This may occur within the tracking module 138, tocontinually augment the information displayed on the visualizationplatform 120. This continual augmentation occurs automatically, as theuser moves and/or manipulates the visualization platform 120 in somemanner, whether it is physical, tactile, or verbal. As additionalinformation is captured and obtained by the framework 100, it isrendered for display on the visualization platform in modules 134 and136, and then presented for display to augment the data already beingdisplayed at step 280. Module 136 may modify and configure thisadditional information by one or more of filtering, ordering, orotherwise transforming it, particularly in response to user manipulationof the visualization platform 120.

As the process 200 continues with this continual augmentation, theframework 100 can also be used to instruct and/or initialize output data150, such as generation of performance maps 151 and responsive actions152, at step 290. As noted above, this output data 150 may take manyforms, but it is to be understood that the framework 100 and process 200may be used to perform many tasks relative to the intersection 104 androadway 106.

The systems and methods of the present invention may be implemented inmany different computing environments. For example, the augmentedreality visualization framework for traffic applications may beimplemented in conjunction with a special purpose computer, a programmedmicroprocessor or microcontroller and peripheral integrated circuitelement(s), an ASIC or other integrated circuit, a digital signalprocessor, electronic or logic circuitry such as discrete elementcircuit, a programmable logic device or gate array such as a PLD, PLA,FPGA, PAL, and any comparable means. In general, any means ofimplementing the methodology illustrated herein can be used to implementthe various aspects of the present invention. Exemplary hardware thatcan be used for the present invention includes computers, handhelddevices, telephones (e.g., cellular, Internet enabled, digital, analog,hybrids, and others), and other such hardware. Some of these devicesinclude processors (e.g., a single or multiple microprocessors), memory,nonvolatile storage, input devices, and output devices. Furthermore,alternative software implementations including, but not limited to,distributed processing, parallel processing, or virtual machineprocessing can also be configured to perform the methods describedherein.

The systems and methods of the present invention may also be partiallyimplemented in software that can be stored on a storage medium, executedon programmed general-purpose computer with the cooperation of acontroller and memory, a special purpose computer, a microprocessor, orthe like. In these instances, the systems and methods of this inventioncan be implemented as a program embedded on personal computer such as anapplet, JAVA® or CGI script, as a resource residing on a server orcomputer workstation, as a routine embedded in a dedicated measurementsystem, system component, or the like. The system can also beimplemented by physically incorporating the system and/or method into asoftware and/or hardware system.

Additionally, the data processing functions disclosed herein may beperformed by one or more program instructions stored in or executed bysuch memory, and further may be performed by one or more modulesconfigured to carry out those program instructions. Modules are intendedto refer to any known or later developed hardware, software, firmware,artificial intelligence, fuzzy logic, expert system or combination ofhardware and software that is capable of performing the data processingfunctionality described herein.

The foregoing descriptions of embodiments of the present invention havebeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Accordingly, many alterations, modifications andvariations are possible in light of the above teachings, may be made bythose having ordinary skill in the art without departing from the spiritand scope of the invention. It is therefore intended that the scope ofthe invention be limited not by this detailed description. For example,notwithstanding the fact that the elements of a claim are set forthbelow in a certain combination, it must be expressly understood that theinvention includes other combinations of fewer, more or differentelements, which are disclosed in above even when not initially claimedin such combinations.

The words used in this specification to describe the invention and itsvarious embodiments are to be understood not only in the sense of theircommonly defined meanings, but to include by special definition in thisspecification structure, material or acts beyond the scope of thecommonly defined meanings. Thus if an element can be understood in thecontext of this specification as including more than one meaning, thenits use in a claim must be understood as being generic to all possiblemeanings supported by the specification and by the word itself.

The definitions of the words or elements of the following claims are,therefore, defined in this specification to include not only thecombination of elements which are literally set forth, but allequivalent structure, material or acts for performing substantially thesame function in substantially the same way to obtain substantially thesame result. In this sense it is therefore contemplated that anequivalent substitution of two or more elements may be made for any oneof the elements in the claims below or that a single element may besubstituted for two or more elements in a claim. Although elements maybe described above as acting in certain combinations and even initiallyclaimed as such, it is to be expressly understood that one or moreelements from a claimed combination can in some cases be excised fromthe combination and that the claimed combination may be directed to asub-combination or variation of a sub-combination.

Insubstantial changes from the claimed subject matter as viewed by aperson with ordinary skill in the art, now known or later devised, areexpressly contemplated as being equivalently within the scope of theclaims. Therefore, obvious substitutions now or later known to one withordinary skill in the art are defined to be within the scope of thedefined elements.

The claims are thus to be understood to include what is specificallyillustrated and described above, what is conceptually equivalent, whatcan be obviously substituted and also what essentially incorporates theessential idea of the invention.

The invention claimed is:
 1. A method, comprising: capturing input datarepresenting a physical-world characteristic relative to atransportation environment that includes at least one of anintersection, a roadway, and conditions in the transportationenvironment where traffic activity occurs, the input data including atleast one of a video image, a video stream, or a map of thephysical-world characteristic; rendering the input data on avisualization platform for display of the at least one of a video image,a video stream, or a map of the physical-world characteristic; detectinga geographical position, and one or more of a movement, tilt or a changein direction of the visualization platform; requesting traffic data forthe physical-world characteristic based on the geographical position andthe one or more of a movement, tilt or a change in direction of thevisualization platform; blending the input data and traffic data byoverlaying the traffic data onto the at least one of a video image, avideo stream, or a map of the physical-world characteristic to augmentthe input data with the traffic data on the visualization platform; andcontinually augmenting the input data by modifying the traffic databased on additional input generated by user manipulation of thevisualization platform, to generate an augmented reality visualizationof the physical-world characteristic; and initiating one or moreresponsive actions from the augmented reality visualization of thephysical-world characteristic, the one or more responsive actionsincluding at least one of modification of traffic data detection, and anadjustment of a traffic signal timing.
 2. The method of claim 1, whereinthe visualization platform is a display apparatus worn by a user.
 3. Themethod of claim 1, wherein the one or more of a movement, tilt or achange in direction of the visualization platform results from the usermanipulation of the visualization platform.
 4. The method of claim 1,wherein the user manipulation of the visualization platform includes oneor more of tactile, verbal, and orientational manipulation.
 5. Themethod of claim 4, wherein the one or more responsive actions from thecontinual augmented reality visualization of the physical worldcharacteristic are initiated by the user manipulation of thevisualization platform.
 6. The method of claim 1, wherein the trafficdata includes one or more of traffic activity information, anidentification of one or more vulnerable road users, environmental orsituational areas of interest, points of interest in the physical worldcharacteristic, scouting reports, objects located in the transportationenvironment, sensor data, satellite data, vehicle data that includes oneor more of a classification of a vehicle and a speed of a vehicle,equipment located within the transportation environment, and tracks ofpast movement of equipment in the transportation environment.
 7. Themethod of claim 1, wherein the visualization platform is a displayapparatus that includes at least one of a mobile device, a wearabledevice, a display inside a piece of traffic equipment, a holo-glassimage displayed inside a piece of traffic equipment, a heads-up display,and a three-dimension projection.
 8. The method of claim 1, wherein theinput data is further augmented on the visualization platform with oneor more of flags, markers, icons, floating images, or colorizedthree-dimensional overlays.
 9. The method of claim 1, wherein theconditions in the transportation environment where traffic activityoccurs include one or more of vehicle-related conditions,pavement-related conditions, and weather-related conditions.
 10. Themethod of claim 1, wherein the continually augmenting the input datafurther comprises creating a virtual reality visualization of thephysical-world characteristic with which the user can interact using thevisualization platform, wherein the augmented reality visualization isblended with the virtual reality visualization.
 11. The method of claim1, further comprising generating a multi-period traffic performance mapfor the transportation environment encompassed by the physical-worldcharacteristic.
 12. A method, comprising: ingesting at least one of avideo image, a video stream, or a map as input data representing aphysical-world characteristic relative to a transportation environmentthat includes at least one of an intersection, a roadway, and conditionsin the transportation environment where traffic activity occurs;analyzing the input data in a plurality of data processing moduleswithin a computing environment in which the plurality of data processingmodules are executed in conjunction with at least onespecifically-configured processor, the data processing modulesconfigured to generate an augmented reality visualization of thephysical-world characteristic, by configuring the input data on avisualization platform for display of the at least one a video image, avideo stream, or a map; determining a geographical position of thevisualization platform; identifying a manipulation of the visualizationplatform from one or more of a movement, tilt or a change in direction;requesting traffic data for the physical-world characteristic based onthe geographical position and the manipulation of the visualizationplatform; overlaying the traffic data onto the at least one a videoimage, a video stream, or a map of the physical-world characteristic toaugment the input data with the traffic data on the visualizationplatform; and continually augmenting the input data by modifying thetraffic data based on additional input generated by the manipulation ofthe visualization platform; and generating, as output data, instructionsfor one or more responsive actions from the augmented realityvisualization of the physical-world characteristic, the one or moreresponsive actions including at least one of modification of trafficdata detection, and an adjustment of a traffic signal timing.
 13. Themethod of claim 12, wherein the visualization platform is a displayapparatus worn by a user.
 14. The method of claim 12, wherein themanipulation of the visualization platform includes one or more oftactile, verbal, and orientational manipulation.
 15. The method of claim14, wherein the one or more responsive actions from the continualaugmented reality visualization of the physical-world characteristic areinitiated by the manipulation of the visualization platform.
 16. Themethod of claim 12, wherein the traffic data includes one or more oftraffic activity information, an identification of one or morevulnerable road users, environmental or situational areas of interest,points of interest in the physical world characteristic, scoutingreports, objects located in the transportation environment, sensor data,satellite data, vehicle data that includes one or more of aclassification of a vehicle and a speed of a vehicle, equipment locatedwithin the transportation environment, and tracks of past movement ofequipment in the transportation environment.
 17. The method of claim 12,wherein the visualization platform is a display apparatus that includesat least one of a mobile device, a wearable device, a display inside apiece of traffic equipment, a holo-glass image displayed inside a pieceof traffic equipment, a heads-up display, and a three-dimensionprojection.
 18. The method of claim 12, wherein the input data isfurther augmented on the visualization platform with one or more offlags, markers, icons, floating images, or colorized three-dimensionaloverlays.
 19. The method of claim 12, wherein the conditions in thetransportation environment where traffic activity occurs include one ormore of vehicle-related conditions, pavement-related conditions, andweather-related conditions.
 20. The method of claim 12, wherein thecontinually augmenting the input data further comprises creating avirtual reality visualization of the physical-world characteristic withwhich the user can interact using the visualization platform, whereinthe augmented reality visualization is blended with the virtual realityvisualization.
 21. The method of claim 12, further comprising generatinga multi-period traffic performance map for the transportationenvironment encompassed by the physical-world characteristic.
 22. Anaugmented reality system for traffic applications, comprising: acomputing environment including at least one computer-readable storagemedium having program instructions stored therein and a computerprocessor operable to execute the program instructions to generate anaugmented reality visualization of the physical-world characteristic,the plurality of data processing modules including: a data ingest moduleconfigured to ingest at least one of a video image, a video stream, or amap as input data representing a physical-world characteristic relativeto a transportation environment that includes at least one of anintersection, a roadway, and conditions in the transportationenvironment where traffic activity occurs; a visualization platformconfigured to display the input data and traffic for the physical-worldcharacteristic, the visualization platform including a display apparatusthat includes at least one of a mobile device, a wearable device, adisplay inside a piece of traffic equipment, a holo-glass imagedisplayed inside a piece of traffic equipment, a heads-up display, and athree-dimension projection; a blending module configured to continuallyaugment the input data with the traffic data in the visualizationplatform, by configuring the input data on a visualization platform fordisplay of the at least one a video image, a video stream, or a map;determining a geographical position of the visualization platform;identifying a manipulation of the visualization platform from one ormore of a movement, tilt or a change in direction; requesting thetraffic data for the physical-world characteristic based on thegeographical position and the manipulation of the visualizationplatform; overlaying the traffic data onto the at least one a videoimage, a video stream, or a map of the physical-world characteristic toaugment the input data with the traffic data on the visualizationplatform; and modifying the traffic data based on additional inputgenerated by the manipulation of the visualization platform, wherein thevisualization platform is configured to initiate one or more responsiveactions from the augmented reality visualization of the physical-worldcharacteristic, the one or more responsive actions including at leastone of modification of traffic data detection, and an adjustment of atraffic signal timing.
 23. The system of claim 22, wherein themanipulation of the visualization platform includes one or more oftactile, verbal, and orientational manipulation.
 24. The system of claim23, wherein the one or more responsive actions from the continualaugmented reality visualization of the physical-world characteristic areinitiated by the manipulation of the visualization platform.
 25. Thesystem of claim 22, wherein the traffic data includes one or more oftraffic activity information, an identification of one or morevulnerable road users, environmental or situational areas of interest,points of interest in the physical-world characteristic, scoutingreports, objects located in the transportation environment, sensor data,satellite data, vehicle data that includes one or more of aclassification of a vehicle and a speed of a vehicle, equipment locatedwithin the transportation environment, and tracks of past movement ofequipment in the transportation environment.
 26. The system of claim 22,wherein the input data is further augmented on the visualizationplatform with one or more of flags, markers, icons, floating images, orcolorized three-dimensional overlays.
 27. The system of claim 22,wherein the conditions in the transportation environment where trafficactivity occurs include one or more of vehicle-related conditions,pavement-related conditions, and weather-related conditions.
 28. Thesystem of claim 22, further comprising creating a virtual realityvisualization of the physical-world characteristic with which the usercan interact using the visualization platform, wherein the augmentedreality visualization is blended with the virtual reality visualization.29. The system of claim 22, wherein the visualization platform isfurther configured to generate a multi-period traffic performance mapfor the transportation environment encompassed by the physical-worldcharacteristic.